Purpose: Revenue vs AdSpend effect

Author: Gerardo Gandara

Contact:

Client:

Code created: 2023-10-11

Last updated: 2023-10-11

Source: https://github.com/ggandara13/myrepository

Comment: https://getrecast.com/modern-media-mix-modeling/.

Libraries

These are the libraries to load

pkg <- c('pivottabler', 'tidyverse','tibbletime','anomalize','timetk')
#install.packages(pkg)

library(pivottabler)
library(lubridate)
library(dplyr)
library(ggplot2)
library(tidyverse)
library(tibbletime)
library(anomalize)
library(timetk)
library(forecast)

Read the revenue and AdSpend

We check the date type of the ‘date’ column and change the type to DATE

The CSV data is avaialble in the github repository


setwd("/Users/gerardogandara/Documents/recast/")

## Read file as CSV
df_spend <- read.csv('https://raw.githubusercontent.com/ggandara13/myrepository/master/acme_spend.csv')
df_revenue <- read.csv('https://raw.githubusercontent.com/ggandara13/myrepository/master/acme_revenue.csv')

# Convert char format to DATE format
df_revenue$date <- as.Date(df_revenue$date, format =  "%m/%d/%y")
df_spend$date <- as.Date(df_spend$date, format =  "%m/%d/%y")

# Check the format and content of each dataframe
str(df_revenue)
'data.frame':   1035 obs. of  5 variables:
 $ X              : int  1 2 3 4 5 6 7 8 9 10 ...
 $ date           : Date, format: "2020-01-01" "2020-01-02" "2020-01-03" ...
 $ revenue_dtc    : num  16000 19306 22052 23177 26169 ...
 $ revenue_amazon : num  50000 60332 68914 72428 81778 ...
 $ revenue_walmart: num  34000 41026 46861 49251 55609 ...
str(df_spend)
'data.frame':   10350 obs. of  4 variables:
 $ X      : int  1 2 3 4 5 6 7 8 9 10 ...
 $ date   : Date, format: "2020-01-01" "2020-01-01" "2020-01-01" ...
 $ channel: chr  "facebook_prospecting" "facebook_retargeting" "google_branded_search" "google_nonbranded_search" ...
 $ spend  : num  4154 2679 474 3817 7421 ...

Check there are not NULL values


paste("Total of NULL in Spend:", sum(is.na(df_spend)))
[1] "Total of NULL in Spend: 0"
paste("\nTotal of NULL in Revenue:", sum(is.na(df_revenue)))
[1] "\nTotal of NULL in Revenue: 0"
paste("\n\nSummary\n")
[1] "\n\nSummary\n"
# you can check with summary as well and see the columns
summary(df_spend)
       X              date              channel              spend            Year     
 Min.   :    1   Min.   :2020-01-01   Length:10350       Min.   :    0   Min.   :2020  
 1st Qu.: 2588   1st Qu.:2020-09-15   Class :character   1st Qu.: 1278   1st Qu.:2020  
 Median : 5176   Median :2021-06-01   Mode  :character   Median : 5268   Median :2021  
 Mean   : 5176   Mean   :2021-06-01                      Mean   : 7740   Mean   :2021  
 3rd Qu.: 7763   3rd Qu.:2022-02-15                      3rd Qu.:11649   3rd Qu.:2022  
 Max.   :10350   Max.   :2022-10-31                      Max.   :36401   Max.   :2022  
summary(df_revenue)
       X               date             revenue_dtc     revenue_amazon   revenue_walmart 
 Min.   :   1.0   Min.   :2020-01-01   Min.   :     0   Min.   :     0   Min.   :     0  
 1st Qu.: 259.5   1st Qu.:2020-09-15   1st Qu.: 41643   1st Qu.:130216   1st Qu.: 88547  
 Median : 518.0   Median :2021-06-01   Median : 72051   Median :225161   Median :153109  
 Mean   : 518.0   Mean   :2021-06-01   Mean   : 72741   Mean   :227062   Mean   :154488  
 3rd Qu.: 776.5   3rd Qu.:2022-02-14   3rd Qu.: 97340   3rd Qu.:303352   3rd Qu.:206279  
 Max.   :1035.0   Max.   :2022-10-31   Max.   :193005   Max.   :415952   Max.   :282848  
sum(is.na(df_spend))/nrow(df_spend)*100
[1] 0
sum(is.na(df_revenue))/nrow(df_revenue)*100
[1] 0

Since we have 1,035 rows, we can remove because that represents less than 1% or just replace with zero

Now, let’s replace with zero since it is a time series


df_revenue[is.na(df_revenue)] <- 0
df_spend[is.na(df_spend)] <- 0


cat("Total of NULL in Spend:", sum(is.na(df_spend)))
Total of NULL in Spend: 0
cat("\nTotal of NULL in Revenue:", sum(is.na(df_revenue)))

Total of NULL in Revenue: 0

1- Which channel had the largest increase in spend in 2022 compared to the same date range in 2021?

Using lubridate to filter the year of the date, also use the pivottabler library to group/aggregate it

Answer: online_video

# print the first value of the dataframe
paste("The channel with highest LIFT is:", row.names(df_summary_final)[1] )
[1] "The channel with highest LIFT is: online_video"

2- In terms of total revenue, are there any anomalous days?

Group by day to see the trend and plot


df_revenue_total <- df_revenue %>%
  mutate(total_revenue = ( df_revenue$revenue_dtc + df_revenue$revenue_amazon + df_revenue$revenue_walmart  ))

# Group by sum using R Base aggregate()
agg_df <- aggregate(df_revenue_total$total_revenue, by=list(df_revenue_total$date), FUN=sum)
colnames(agg_df) <- c('date','revenue') 


#Plot sales over 36 months
ggplot(agg_df, aes(x=date, y = revenue)) + geom_line()


paste("Visually we can see some outliers")
[1] "Visually we can see some outliers"

Now, we can use the anomalties library to identify the values

Use time_decompose() to decompose a time series prior to performing anomaly detection with anomalize(). Typically, anomalize() is performed on the “remainder” of the time series decomposition.

The return has three columns: “remainder_l1” (lower limit for anomalies), “remainder_l2” (upper limit for anomalies), and “anomaly” (Yes/No).

We can print the detail of the anomalies, for example the last 4

tail(df_anomalized %>%
  filter(anomaly == 'Yes'),4)
NA

You can also use the timetk package for dynamic plot

agg_df %>% timetk::plot_anomaly_diagnostics(date,revenue, .facet_ncol = 2)
frequency = 7 observations per 1 week
trend = 92 observations per 3 months

3- In which month of the year does Acme tend to make the most revenue?

We can compute month column before we aggregate and sort to get the popular month

Most profitale month in a year is the month number: 3


# Calculate the month column
monthfunction<-function(dataframe, datecolumn) {
    month(dataframe[,datecolumn])
}
agg_df$Month <- monthfunction(agg_df, "date")

# group by month

by_month <- aggregate(agg_df$revenue, by = list(agg_df$Month), FUN = sum)
colnames(by_month) <- c('month','revenue') 



by_month <- by_month[order(by_month$revenue, decreasing = TRUE), ]

# print the first value of the dataframe
print(by_month)
paste("Most profitale month in a year is the month number:", row.names(by_month)[1] )
[1] "Most profitale month in a year is the month number: 3"

4- Does Acme’s marketing spend tend to follow a similar pattern to revenue?

We need to merge both dataframes to have the TS of Revenue using AdSped as regressor Convert the dataframe to TS then plot, then plot the results

Visually we can confirm they have a similar pattern.


#select the date and total_revenue of Acme
df_only_revenue <- select(df_revenue_total, date, total_revenue)

#aggregate spend of all media
df_spend_total <- aggregate(df_spend$spend, by=list(df_spend$date), FUN=sum)
colnames(df_spend_total) <- c('date','spend') 

df_rev_spend <- merge(df_only_revenue, df_spend_total, by = "date")


#Convert dataframe to time series object using the ts() function
acme_ts <- ts(data = df_rev_spend[,c(2,3)])


# Time plot of both variables
autoplot(acme_ts, facets = TRUE)

LS0tCnRpdGxlOiAiQXNzaWdubWVudC1SRUNBU1QgTW9kZXJuIE1hcmtldGluZyBNaXggTW9kZWxpbmcgIgpvdXRwdXQ6CiAgaHRtbF9kb2N1bWVudDoKICAgIGRmX3ByaW50OiBwYWdlZAotLS0KCiMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwojICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICMKIyBQdXJwb3NlOiAgICAgICBSZXZlbnVlIHZzIEFkU3BlbmQgZWZmZWN0ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAjCiMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIwojIEF1dGhvcjogICAgICAgIEdlcmFyZG8gR2FuZGFyYSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICMKIyBDb250YWN0OiAgICAgICBnZXJhcmRvLmdhbmRhcmFAZ21haWwuY29tICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAjCiMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIwojIENsaWVudDogICAgICAgIGFuZHlAZ2V0cmVjYXN0LmNvbSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICMKIyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAjCiMgQ29kZSBjcmVhdGVkOiAgMjAyMy0xMC0xMSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIwojIExhc3QgdXBkYXRlZDogIDIwMjMtMTAtMTEgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICMKIyBTb3VyY2U6ICAgICAgICBodHRwczovL2dpdGh1Yi5jb20vZ2dhbmRhcmExMy9teXJlcG9zaXRvcnkgICAgICAgICAgICAgICAjCiMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIwojIENvbW1lbnQ6ICAgICAgIGh0dHBzOi8vZ2V0cmVjYXN0LmNvbS9tb2Rlcm4tbWVkaWEtbWl4LW1vZGVsaW5nLy4gICAgICAgICMKIyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAjCiMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwoKCiMgTGlicmFyaWVzClRoZXNlIGFyZSB0aGUgbGlicmFyaWVzIHRvIGxvYWQKCgpgYGB7cn0KcGtnIDwtIGMoJ3Bpdm90dGFibGVyJywgJ3RpZHl2ZXJzZScsJ3RpYmJsZXRpbWUnLCdhbm9tYWxpemUnLCd0aW1ldGsnKQojaW5zdGFsbC5wYWNrYWdlcyhwa2cpCgpsaWJyYXJ5KHBpdm90dGFibGVyKQpsaWJyYXJ5KGx1YnJpZGF0ZSkKbGlicmFyeShkcGx5cikKbGlicmFyeShnZ3Bsb3QyKQpsaWJyYXJ5KHRpZHl2ZXJzZSkKbGlicmFyeSh0aWJibGV0aW1lKQpsaWJyYXJ5KGFub21hbGl6ZSkKbGlicmFyeSh0aW1ldGspCmxpYnJhcnkoZm9yZWNhc3QpCgpgYGAKCgoKUmVhZCB0aGUgcmV2ZW51ZSBhbmQgQWRTcGVuZAoKV2UgY2hlY2sgdGhlIGRhdGUgdHlwZSBvZiB0aGUgJ2RhdGUnIGNvbHVtbiBhbmQgY2hhbmdlIHRoZSB0eXBlIHRvIERBVEUKClRoZSBDU1YgZGF0YSBpcyBhdmFpYWxibGUgaW4gdGhlIGdpdGh1YiByZXBvc2l0b3J5CgoKYGBge3J9CgpzZXR3ZCgiL1VzZXJzL2dlcmFyZG9nYW5kYXJhL0RvY3VtZW50cy9yZWNhc3QvIikKCiMjIFJlYWQgZmlsZSBhcyBDU1YKZGZfc3BlbmQgPC0gcmVhZC5jc3YoJ2h0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9nZ2FuZGFyYTEzL215cmVwb3NpdG9yeS9tYXN0ZXIvYWNtZV9zcGVuZC5jc3YnKQpkZl9yZXZlbnVlIDwtIHJlYWQuY3N2KCdodHRwczovL3Jhdy5naXRodWJ1c2VyY29udGVudC5jb20vZ2dhbmRhcmExMy9teXJlcG9zaXRvcnkvbWFzdGVyL2FjbWVfcmV2ZW51ZS5jc3YnKQoKIyBDb252ZXJ0IGNoYXIgZm9ybWF0IHRvIERBVEUgZm9ybWF0CmRmX3JldmVudWUkZGF0ZSA8LSBhcy5EYXRlKGRmX3JldmVudWUkZGF0ZSwgZm9ybWF0ID0gICIlbS8lZC8leSIpCmRmX3NwZW5kJGRhdGUgPC0gYXMuRGF0ZShkZl9zcGVuZCRkYXRlLCBmb3JtYXQgPSAgIiVtLyVkLyV5IikKCiMgQ2hlY2sgdGhlIGZvcm1hdCBhbmQgY29udGVudCBvZiBlYWNoIGRhdGFmcmFtZQpzdHIoZGZfcmV2ZW51ZSkKc3RyKGRmX3NwZW5kKQoKYGBgCgoKQ2hlY2sgdGhlcmUgYXJlIG5vdCBOVUxMIHZhbHVlcyAKCmBgYHtyfQoKcGFzdGUoIlRvdGFsIG9mIE5VTEwgaW4gU3BlbmQ6Iiwgc3VtKGlzLm5hKGRmX3NwZW5kKSkpCgpwYXN0ZSgiXG5Ub3RhbCBvZiBOVUxMIGluIFJldmVudWU6Iiwgc3VtKGlzLm5hKGRmX3JldmVudWUpKSkKCnBhc3RlKCJcblxuU3VtbWFyeVxuIikKIyB5b3UgY2FuIGNoZWNrIHdpdGggc3VtbWFyeSBhcyB3ZWxsIGFuZCBzZWUgdGhlIGNvbHVtbnMKc3VtbWFyeShkZl9zcGVuZCkKc3VtbWFyeShkZl9yZXZlbnVlKQoKc3VtKGlzLm5hKGRmX3NwZW5kKSkvbnJvdyhkZl9zcGVuZCkqMTAwCnN1bShpcy5uYShkZl9yZXZlbnVlKSkvbnJvdyhkZl9yZXZlbnVlKSoxMDAKCgpgYGAKU2luY2Ugd2UgaGF2ZSAxLDAzNSByb3dzLCB3ZSBjYW4gcmVtb3ZlIGJlY2F1c2UgdGhhdCByZXByZXNlbnRzIGxlc3MgdGhhbiAxJSBvciBqdXN0IHJlcGxhY2Ugd2l0aCB6ZXJvCgpOb3csIGxldCdzIHJlcGxhY2Ugd2l0aCB6ZXJvIHNpbmNlIGl0IGlzIGEgdGltZSBzZXJpZXMKYGBge3J9CgpkZl9yZXZlbnVlW2lzLm5hKGRmX3JldmVudWUpXSA8LSAwCmRmX3NwZW5kW2lzLm5hKGRmX3NwZW5kKV0gPC0gMAoKCmNhdCgiVG90YWwgb2YgTlVMTCBpbiBTcGVuZDoiLCBzdW0oaXMubmEoZGZfc3BlbmQpKSkKY2F0KCJcblRvdGFsIG9mIE5VTEwgaW4gUmV2ZW51ZToiLCBzdW0oaXMubmEoZGZfcmV2ZW51ZSkpKQoKCmBgYAoKCgoKIyMgMS0gV2hpY2ggY2hhbm5lbCBoYWQgdGhlIGxhcmdlc3QgaW5jcmVhc2UgaW4gc3BlbmQgaW4gMjAyMiBjb21wYXJlZCB0byB0aGUgc2FtZSBkYXRlIHJhbmdlIGluIDIwMjE/CgpVc2luZyBsdWJyaWRhdGUgdG8gZmlsdGVyIHRoZSB5ZWFyIG9mIHRoZSBkYXRlLCBhbHNvIHVzZSB0aGUgcGl2b3R0YWJsZXIgbGlicmFyeSB0byBncm91cC9hZ2dyZWdhdGUgaXQKCkFuc3dlcjogb25saW5lX3ZpZGVvCgoKYGBge3J9CnllYXJmdW5jdGlvbjwtZnVuY3Rpb24oZGF0YWZyYW1lLCBkYXRlY29sdW1uKSB7CiAgICB5ZWFyKGRhdGFmcmFtZVssZGF0ZWNvbHVtbl0pCn0KZGZfc3BlbmQkWWVhciA8LSB5ZWFyZnVuY3Rpb24oZGZfc3BlbmQsICJkYXRlIikKCgpkZiA8LSBkZl9zcGVuZCAlPiUgZ3JvdXBfYnkoY2hhbm5lbCkgJT4lIAogIGZpbHRlcihsdWJyaWRhdGU6OnllYXIoZGF0ZSkgJWluJSBjKDIwMjEsIDIwMjIpICkKZGYgPC0gYXMuZGF0YS5mcmFtZShkZikKCgojIGluaXRpYWxpY2UgdGhlIFBJVk9UCnB0IDwtIFBpdm90VGFibGUkbmV3KCkKIyBhZGQgdGhlIGRmIHdpdGggMiB5ZWFycwpwdCRhZGREYXRhKGRmKQpwdCRhZGRDb2x1bW5EYXRhR3JvdXBzKCJZZWFyIikKcHQkYWRkUm93RGF0YUdyb3VwcygiY2hhbm5lbCIpCnB0JGRlZmluZUNhbGN1bGF0aW9uKGNhbGN1bGF0aW9uTmFtZT0iVG90YWxTcGVuZCIsIHN1bW1hcmlzZUV4cHJlc3Npb249InN1bShzcGVuZCkiKQpwdCRldmFsdWF0ZVBpdm90KCkKCiNjb3ZuZXJ0IHRvIGRhdGFmcmFtZQpkZjEgPC0gcHQkYXNEYXRhRnJhbWUoKQpkZl9zdW1tYXJ5X2ZpbmFsIDwtIGRmMSAlPiUKICBtdXRhdGUoaW5kZXhfbGlmdCA9ICggKGRmMVssMl0vIGRmMVssMV0pIC0gMSApICoxMDAgICkKCmRmX3N1bW1hcnlfZmluYWw8LSBkZl9zdW1tYXJ5X2ZpbmFsW29yZGVyKGRmX3N1bW1hcnlfZmluYWwkaW5kZXhfbGlmdCwgZGVjcmVhc2luZyA9IFRSVUUpLCBdCgoKIyBwcmludCB0aGUgZmlyc3QgdmFsdWUgb2YgdGhlIGRhdGFmcmFtZQpwYXN0ZSgiVGhlIGNoYW5uZWwgd2l0aCBoaWdoZXN0IExJRlQgaXM6Iiwgcm93Lm5hbWVzKGRmX3N1bW1hcnlfZmluYWwpWzFdICkKCgpgYGAKCgoKCiMjIDItIEluIHRlcm1zIG9mIHRvdGFsIHJldmVudWUsIGFyZSB0aGVyZSBhbnkgYW5vbWFsb3VzIGRheXM/CgpHcm91cCBieSBkYXkgdG8gc2VlIHRoZSB0cmVuZCBhbmQgcGxvdAoKCmBgYHtyfQoKZGZfcmV2ZW51ZV90b3RhbCA8LSBkZl9yZXZlbnVlICU+JQogIG11dGF0ZSh0b3RhbF9yZXZlbnVlID0gKCBkZl9yZXZlbnVlJHJldmVudWVfZHRjICsgZGZfcmV2ZW51ZSRyZXZlbnVlX2FtYXpvbiArIGRmX3JldmVudWUkcmV2ZW51ZV93YWxtYXJ0ICApKQoKIyBHcm91cCBieSBzdW0gdXNpbmcgUiBCYXNlIGFnZ3JlZ2F0ZSgpCmFnZ19kZiA8LSBhZ2dyZWdhdGUoZGZfcmV2ZW51ZV90b3RhbCR0b3RhbF9yZXZlbnVlLCBieT1saXN0KGRmX3JldmVudWVfdG90YWwkZGF0ZSksIEZVTj1zdW0pCmNvbG5hbWVzKGFnZ19kZikgPC0gYygnZGF0ZScsJ3JldmVudWUnKSAKCgojUGxvdCBzYWxlcyBvdmVyIDM2IG1vbnRocwpnZ3Bsb3QoYWdnX2RmLCBhZXMoeD1kYXRlLCB5ID0gcmV2ZW51ZSkpICsgZ2VvbV9saW5lKCkKCnBhc3RlKCJWaXN1YWxseSB3ZSBjYW4gc2VlIHNvbWUgb3V0bGllcnMiKQpgYGAKCk5vdywgd2UgY2FuIHVzZSB0aGUgYW5vbWFsdGllcyBsaWJyYXJ5IHRvIGlkZW50aWZ5IHRoZSB2YWx1ZXMKClVzZSB0aW1lX2RlY29tcG9zZSgpIHRvIGRlY29tcG9zZSBhIHRpbWUgc2VyaWVzIHByaW9yIHRvIHBlcmZvcm1pbmcgYW5vbWFseSBkZXRlY3Rpb24gd2l0aCBhbm9tYWxpemUoKS4gVHlwaWNhbGx5LCBhbm9tYWxpemUoKSBpcyBwZXJmb3JtZWQgb24gdGhlICJyZW1haW5kZXIiIG9mIHRoZSB0aW1lIHNlcmllcyBkZWNvbXBvc2l0aW9uLgoKVGhlIHJldHVybiBoYXMgdGhyZWUgY29sdW1uczogInJlbWFpbmRlcl9sMSIgKGxvd2VyIGxpbWl0IGZvciBhbm9tYWxpZXMpLCAicmVtYWluZGVyX2wyIiAodXBwZXIgbGltaXQgZm9yIGFub21hbGllcyksIGFuZCAiYW5vbWFseSIgKFllcy9ObykuCgoKCmBgYHtyfQojIENvbnZlcnQgZGYgdG8gYSB0aWJibGUKZGYgPC0gYXNfdGliYmxlKGFnZ19kZikKY2xhc3MoZGYpCgpkZl9hbm9tYWxpemVkIDwtIGRmICU+JQogICAgdGltZV9kZWNvbXBvc2UocmV2ZW51ZSwgbWV0aG9kID0gInN0bCIsIG1lcmdlID0gVFJVRSkgJT4lCiAgICBhbm9tYWxpemUocmVtYWluZGVyLCBtZXRob2QgPSAiaXFyIiwgYWxwaGEgPSAwLjA1LCBtYXhfYW5vbXMgPSAwLjIpICU+JQogICAgdGltZV9yZWNvbXBvc2UoKQoKI1dlIGNhbiB0aGVuIHZpc3VhbGl6ZSB0aGUgYW5vbWFsaWVzIHVzaW5nIHRoZSBwbG90X2Fub21hbGllcygpIGZ1bmN0aW9uLiAKZGZfYW5vbWFsaXplZCAlPiUgcGxvdF9hbm9tYWxpZXMobmNvbCA9IDEsIGFscGhhX2RvdHMgPSAwLjc1KQoKYGBgCgpXZSBjYW4gcHJpbnQgdGhlIGRldGFpbCBvZiB0aGUgYW5vbWFsaWVzLCBmb3IgZXhhbXBsZSB0aGUgbGFzdCA0CgoKYGBge3J9CnRhaWwoZGZfYW5vbWFsaXplZCAlPiUKICBmaWx0ZXIoYW5vbWFseSA9PSAnWWVzJyksNCkKYGBgCgpZb3UgY2FuIGFsc28gdXNlIHRoZSB0aW1ldGsgcGFja2FnZSBmb3IgZHluYW1pYyBwbG90CgpgYGB7cn0KYWdnX2RmICU+JSB0aW1ldGs6OnBsb3RfYW5vbWFseV9kaWFnbm9zdGljcyhkYXRlLHJldmVudWUsIC5mYWNldF9uY29sID0gMikKCmBgYAoKCiMjIDMtIEluIHdoaWNoIG1vbnRoIG9mIHRoZSB5ZWFyIGRvZXMgQWNtZSB0ZW5kIHRvIG1ha2UgdGhlIG1vc3QgcmV2ZW51ZT8KCldlIGNhbiBjb21wdXRlIG1vbnRoIGNvbHVtbiBiZWZvcmUgd2UgYWdncmVnYXRlIGFuZCBzb3J0IHRvIGdldCB0aGUgcG9wdWxhciBtb250aAoKTW9zdCBwcm9maXRhbGUgbW9udGggaW4gYSB5ZWFyIGlzIHRoZSBtb250aCBudW1iZXI6IDMKCmBgYHtyfQoKIyBDYWxjdWxhdGUgdGhlIG1vbnRoIGNvbHVtbgptb250aGZ1bmN0aW9uPC1mdW5jdGlvbihkYXRhZnJhbWUsIGRhdGVjb2x1bW4pIHsKICAgIG1vbnRoKGRhdGFmcmFtZVssZGF0ZWNvbHVtbl0pCn0KYWdnX2RmJE1vbnRoIDwtIG1vbnRoZnVuY3Rpb24oYWdnX2RmLCAiZGF0ZSIpCgojIGdyb3VwIGJ5IG1vbnRoCgpieV9tb250aCA8LSBhZ2dyZWdhdGUoYWdnX2RmJHJldmVudWUsIGJ5ID0gbGlzdChhZ2dfZGYkTW9udGgpLCBGVU4gPSBzdW0pCmNvbG5hbWVzKGJ5X21vbnRoKSA8LSBjKCdtb250aCcsJ3JldmVudWUnKSAKCgoKYnlfbW9udGggPC0gYnlfbW9udGhbb3JkZXIoYnlfbW9udGgkcmV2ZW51ZSwgZGVjcmVhc2luZyA9IFRSVUUpLCBdCgojIHByaW50IHRoZSBmaXJzdCB2YWx1ZSBvZiB0aGUgZGF0YWZyYW1lCnByaW50KGJ5X21vbnRoKQpwYXN0ZSgiTW9zdCBwcm9maXRhbGUgbW9udGggaW4gYSB5ZWFyIGlzIHRoZSBtb250aCBudW1iZXI6Iiwgcm93Lm5hbWVzKGJ5X21vbnRoKVsxXSApCgpgYGAKCgojIyA0LSBEb2VzIEFjbWUncyBtYXJrZXRpbmcgc3BlbmQgdGVuZCB0byBmb2xsb3cgYSBzaW1pbGFyIHBhdHRlcm4gdG8gcmV2ZW51ZT8KCldlIG5lZWQgdG8gbWVyZ2UgYm90aCBkYXRhZnJhbWVzIHRvIGhhdmUgdGhlIFRTIG9mIFJldmVudWUgdXNpbmcgQWRTcGVkIGFzIHJlZ3Jlc3NvcgpDb252ZXJ0IHRoZSBkYXRhZnJhbWUgdG8gVFMgdGhlbiBwbG90LCB0aGVuIHBsb3QgdGhlIHJlc3VsdHMKClZpc3VhbGx5IHdlIGNhbiBjb25maXJtIHRoZXkgaGF2ZSBhIHNpbWlsYXIgcGF0dGVybi4KCmBgYHtyfQoKI3NlbGVjdCB0aGUgZGF0ZSBhbmQgdG90YWxfcmV2ZW51ZSBvZiBBY21lCmRmX29ubHlfcmV2ZW51ZSA8LSBzZWxlY3QoZGZfcmV2ZW51ZV90b3RhbCwgZGF0ZSwgdG90YWxfcmV2ZW51ZSkKCiNhZ2dyZWdhdGUgc3BlbmQgb2YgYWxsIG1lZGlhCmRmX3NwZW5kX3RvdGFsIDwtIGFnZ3JlZ2F0ZShkZl9zcGVuZCRzcGVuZCwgYnk9bGlzdChkZl9zcGVuZCRkYXRlKSwgRlVOPXN1bSkKY29sbmFtZXMoZGZfc3BlbmRfdG90YWwpIDwtIGMoJ2RhdGUnLCdzcGVuZCcpIAoKZGZfcmV2X3NwZW5kIDwtIG1lcmdlKGRmX29ubHlfcmV2ZW51ZSwgZGZfc3BlbmRfdG90YWwsIGJ5ID0gImRhdGUiKQoKCiNDb252ZXJ0IGRhdGFmcmFtZSB0byB0aW1lIHNlcmllcyBvYmplY3QgdXNpbmcgdGhlIHRzKCkgZnVuY3Rpb24KYWNtZV90cyA8LSB0cyhkYXRhID0gZGZfcmV2X3NwZW5kWyxjKDIsMyldKQoKCiMgVGltZSBwbG90IG9mIGJvdGggdmFyaWFibGVzCmF1dG9wbG90KGFjbWVfdHMsIGZhY2V0cyA9IFRSVUUpCgpgYGAKCgoKCgoKIyMgNS0gQmFzZWQgb24gd2hhdCB5b3Ugc2VlIGhlcmUsIGNvdWxkIHlvdSBtYWtlIHRoZSBjYXNlIHRoYXQgbWFya2V0aW5nIHNwZW5kIGlzIGNhdXNhbGx5IHJlbGF0ZWQgdG8gcmV2ZW51ZT8gV2h5IG9yIHdoeSBub3Q/CgpWaXN1YWxseSB3ZSBjYW4gc2VlIHRoZSBlZmZlY3QgaGFzIGhpZ2ggY29ycmVsYXRpb24sIGFuZCB3aGVuIHdlIGNhbGN1bGF0ZSB0aGUgUjIgaXMgMC45MSB3aXRoIHA9dmFsdWUgYmFzaWNhbGx5IHplcm8uIFNvLCBpdCBpcyBzdGF0aXNjYWxseSBjb3JyZWxhdGVkLgoKCmBgYHtyfQoKbGlicmFyeSgiZ2dwdWJyIikKCmRmX3Jldl9zcGVuZAoKZ2dzY2F0dGVyKGRmX3Jldl9zcGVuZCwgeCA9ICJzcGVuZCIsIHkgPSAidG90YWxfcmV2ZW51ZSIsIAogICAgICAgICAgYWRkID0gInJlZy5saW5lIiwgY29uZi5pbnQgPSBUUlVFLCAKICAgICAgICAgIGNvci5jb2VmID0gVFJVRSwgY29yLm1ldGhvZCA9ICJwZWFyc29uIiwKICAgICAgICAgIHhsYWIgPSAiVG90YWwgQWRTcGVuZCIsIHlsYWIgPSAiVG90YWwgUmV2ZW51ZSAtIDMgc291cmNlcyAiKQoKYGBgCklzIHRoZSByZWxhdGlvbiBsaW5lYXI/IFllcywgZm9ybSB0aGUgcGxvdCBhYm92ZSwgdGhlIHJlbGF0aW9uc2hpcCBpcyBsaW5lYXIuCgpub3cgd2UganN1dCBuZWVkIHRvIHZlcmlmeSB0aGF0IHRoZSBkYXRhIGZyb20gZWFjaCBvZiB0aGUgMiB2YXJpYWJsZXMgKHJldmVudWUsIHNwZW5kKSBmb2xsb3cgYSBub3JtYWwgZGlzdHJpYnV0aW9uLgoKYGBge3J9CgpkZl9yZXZfc3BlbmQKCgojIFNoYXBpcm8tV2lsayBub3JtYWxpdHkgdGVzdCBmb3IgcmV2ZW51ZQpzaGFwaXJvLnRlc3QoZGZfcmV2X3NwZW5kJHRvdGFsX3JldmVudWUpICMgPT4gcC12YWx1ZSA8IDIuMmUtMTYKCiMgU2hhcGlyby1XaWxrIG5vcm1hbGl0eSB0ZXN0IGZvciBzcGVuZApzaGFwaXJvLnRlc3QoZGZfcmV2X3NwZW5kJHNwZW5kKSAjID0+IHAtdmFsdWUgPCAyLjJlLTE2CgoKIyByZXZlbnVlIApnZ3FxcGxvdChkZl9yZXZfc3BlbmQkdG90YWxfcmV2ZW51ZSwgeWxhYiA9ICJyZXZlbnVlIikKIyBzcGVuZApnZ3FxcGxvdChkZl9yZXZfc3BlbmQkc3BlbmQsIHlsYWIgPSAic3BlbmQiKQpgYGAKClNpbmNlIHRoZSBwLXZhbHVlIGlzIHByYWN0aWNhbGx5IHplcm8sIHdlIGNhbiByZWplY3QgdGhlIG51bGwgaHlwb3RoZXNpcyB0aGF0IGl0IGlzIG5vcm1hbCBkaXN0cmlidXRlZCBib3RoIHZhcmlhYmxlcwoKc28gd2UgY2FuIHVzZSBhbm90aGVyIHRlc3QgdG8gY2hlY2sgYXNvY2lhdGlvbgpLZW5kYWxsIHJhbmsgY29ycmVsYXRpb24gdGVzdCAtIEtlbmRhbGzigJlzIHRhdSAKClRoaXMgaXMgdXNlZCB0byBlc3RpbWF0ZSBhIHJhbmstYmFzZWQgbWVhc3VyZSBvZiBhc3NvY2lhdGlvbi4gVGhpcyB0ZXN0IG1heSBiZSB1c2VkIGlmIHRoZSBkYXRhIGRvIG5vdCBuZWNlc3NhcmlseSBjb21lIGZyb20gYSBiaXZhcmlhdGUgbm9ybWFsIGRpc3RyaWJ1dGlvbi4KClRoZSBjb3JyZWxhdGlvbiBjb2VmZmljaWVudCBiZXR3ZWVuIHJldmVudWUgYW5kIHNwZW5kIGFyZSAwLjcyMDkyNDMgYW5kIHRoZSBwLXZhbHVlIDwgMi4yZS0xNi4KRXhpc3QgY29ycmVsYXRpb24KCmJ1dCB3ZSBrbm93IGl0IGlzIG5vdCBjYXVzYXRpb24KCmBgYHtyfQpyZXMyIDwtIGNvci50ZXN0KGRmX3Jldl9zcGVuZCR0b3RhbF9yZXZlbnVlLCBkZl9yZXZfc3BlbmQkc3BlbmQsICBtZXRob2Q9ImtlbmRhbGwiKQpyZXMyCmBgYAoKCldlIGtub3cgdGhhdCB0aGV5IGFyZSBoaWdobHkgY29ycmVsYXRlZC4gV2UgY2FuIHVzZSB0aW1lIHNlcmllcyB0byBzZWUgdGhlIG1hZ25pdHVkZSBvZiB0aGUgZWZmZWN0IG9mIG1hcmtldGluZyB1c2luZyBpdCBhcyByZWdyZXNzb3IKClZpc3VhbGx5IGluIHRoZSBwcmV2aW91cyBwbG90IG9mIHJldmVudWUsIHdlIGNhbiBzZWUgaXQgaXMgc3RhdGlvbmFyeSwgYW5kIHRoZXJlIGlzIHNlYXNvbmFsaXR5LiBXZSBjYW4gdXNlIGF1dG8uYXJpbWEgdG8gY2FsY3VsYXRlIHRoZSByZWdyZXNzb3IgZmFjdG9yCgpUaGlzIGlzIGEgdmVyeSBiYXNpYyBhbmFseXNpcywgYnV0IGl0IGhlbHBzIHRvIHVuZGVyc3RhbmQgaW4gb3ZlcmFsbCBob3cgdGhlIG1hcmtldGluZyBlZmZvcnRzIGFyZSBjcmVhdGluZyBhbiBlZmZlY3QgaW4gdGhlIHJldmVudWUuCgoxLjQ4OTA3MSBpcyB0aGUgcmVncmVzc29yIGZhY3RvcgoKIyMjIyBJbnRlcnByZXRhdGlvbiBvZiBzYWxlcyBpbmNyZWFzZTogRm9yIGV2ZXJ5ICQxLDAwMCAodGhlIHVuaXQgb2YgYWQgc3BlbmQgaXMgaW4gdGhvdXNhbmRzIFVTRCkgaW5jcmVhc2UgaW4gYWR2ZXJ0aXNpbmcgc3BlbmQsIHRoZSB1bml0IHNhbGVzIGluY3JlYXNlIGJ5IDEuNDg5MSAtIEl0IGlzIGEgcG9zaXRpdmUgZWZmZWN0LCBzbyBBQ01FIHByb2JhYmx5IGlzIHVzaW5nIFJFQ0FTVCBhbmQgdGhlIG1hcmtldGluZyBjYW1wYWlnbiBpcyBwZXJmb3JtYWluZyB3ZWxsCgoKCmBgYHtyfQojIHdlIG5lZWQgdG8gY29udmVydCB0aGUgZGFpbHkgZGF0YSB0byB3ZWVrbHkKd2Vla25vIDwtIGFzLm51bWVyaWMoZGZfcmV2X3NwZW5kJGRhdGUgLSBkZl9yZXZfc3BlbmQkZGF0ZVsxXSkgJS8lIDcKV2VlayA8LSBkZl9yZXZfc3BlbmQkZGF0ZVsxXSArIDcgKiB3ZWVrbm8KZGZfYnlfd2VlayA8LSBhZ2dyZWdhdGUoY2JpbmQodG90YWxfcmV2ZW51ZSxzcGVuZCkgIH4gV2VlaywgZGZfcmV2X3NwZW5kLCBzdW0pCgojdG8gY2hlY2sgaXQgbG9va3MgZ29vZApkZl9ieV93ZWVrCgojQ29udmVydCBkYXRhZnJhbWUgdG8gdGltZSBzZXJpZXMgb2JqZWN0IHVzaW5nIHRoZSB0cygpIGZ1bmN0aW9uCmFjbWVfdHNfd2VlayA8LSB0cyhkYXRhID0gZGZfYnlfd2Vla1ssYygyLDMpXSkKIyBUaW1lIHBsb3Qgb2YgYm90aCB2YXJpYWJsZXMKYXV0b3Bsb3QoYWNtZV90c193ZWVrLCBmYWNldHMgPSBUUlVFKQoKIyBGaXQgQVJJTUEgbW9kZWwKZml0IDwtIGF1dG8uYXJpbWEoYWNtZV90c193ZWVrWywgInRvdGFsX3JldmVudWUiXSwgeHJlZyA9IGFjbWVfdHNfd2Vla1ssICJzcGVuZCJdLCAKICAgICAgICAgICAgICAgICAgc3RhdGlvbmFyeSA9IFRSVUUsIHNlYXNvbmFsPVRSVUUsIAogICAgICAgICAgICAgICAgICBhbGxvd2RyaWZ0ID0gRkFMU0UsIGFsbG93bWVhbiA9IEZBTFNFICkjLGxhbWJkYSA9IExhbWJkYSkKCmZpdAoKI1RoZSBhdXRvLmFyaW1hIGZ1bmN0aW9uIGhhcyBmaXQgYSBsaW5lYXIgcmVncmVzc2lvbiBtb2RlbCB0byB0aGUgYWR2ZXJ0aXNpbmcgdmFyaWFibGUgYW5kIGFuIGFuIEFSSU1BIG1vZGVsIHRvIHRoZSB0aW1lIHNlcmllcyAoZGF0ZSkgdmFyaWFibGUuIEFSSU1BKDIsMCwxKQoKI1doYXQgaXMgdGhlIGluY3JlYXNlIGluIHNhbGVzIGZvciBlYWNoIHVuaXQgaW5jcmVhc2UgaW4gYWR2ZXJ0aXNpbmc/CnNhbGVzX2luY3JlYXNlIDwtIGNvZWZmaWNpZW50cyhmaXQpWzRdCnNhbGVzX2luY3JlYXNlCgoKCgojQ2hlY2sgUmVzaWR1YWxzIHRvIHZlcmlmeSB0aGF0IHRoZSByZXNpZHVhbHMgYXJlIHdoaXRlIG5vaXNlIGFuZCBub3JtYWwgZGlzdHJpYnV0ZWQgLSBJbmRlZWQgdGhleSBhcmUgc28gdGhlIG1vZGVsIGlzIGFjY2VwdGFibGUKY2hlY2tyZXNpZHVhbHMoZml0KQoKYGBgCgoKCg==